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DEVICE FOR AGGREGATING, 
TRANSLATING, AND DISSEMINATING 
COMMUNICATIONS WITHIN A MULTIPLE 
DEVICE ENVIRONMENT 

TECHNICAL FIELD 

The present invention relates to electronic device inter- 
actions. More specifically, the present invention relates to an 
aggregator of device interaction for an environment. 

BACKGROUND 

Electronic devices such as household appliances, audio- 
video equipment, computers, and telephones operate within 
a given environment such as the home of a user. However, 
these devices function independently of one another. The 
user must initiate actions on the devices to cause the devices 
to change to a particular state of operation to thereby 
perform a function desired by the user. 

Often, the state of one or more of the electronic devices 
is related to the state of one or more other electronic devices 
within the same environment. For example, a user may be 
watching television (TV) when the telephone rings. The user 
wishes to answer the call, but to effectively communicate 
with the caller, the user must mute the television so that 
sound from the TV does not interfere with the telephone 
conversation. Every time a telephone call is to be answered 
while the user watches TV, the user must again repeat the 
muting process. For each call, once the user hangs up the 
phone, the TV must be manually unmuted so that the user 
can once again listen to the TV program being watched. 

The TV — telephone scenario discussed above is only one 
example. There is an undeterminable number of scenarios 
and devices involved within a given environment. In each 
scenario, the devices do not communicate with one another 
and do not coordinate activities, and as a result the user is 
overly burdened. The number of electronic devices for a 
household is continually increasing, and the resulting burden 
on the user to manually coordinate states of the devices for 
given scenarios is increasing as well. 

To address this problem, devices can be configured with 
communication abilities so that they can communicate with 
one another when one or more devices experience a user 
driven state change. However, when several devices are 
involved in the interaction, various complexities are intro- 
duced. As devices and associated interactions are added to 
the environment, communication paths begin to multiply 
no n- linearly necessitating wider bandwidths to accommo- 
date the increase. Interaction rules become ambiguous 
because devices down a chain of interaction cannot deter- 
mine a proper state change due to several degrees of removal 
from the user driven event. 

Additionally, many different transports are available for 
communication between devices. Absent a standard speci- 
fying a transport for an environment, any transport may be 
chosen as the transport for a particular device. If two devices 
are not equipped with the same transports, then communi- 
cation between the devices becomes impossible. These 
devices will not satisfactorily coexist and interact in an 
environment. 

Therefore, there is a need for aggregating communication 
paths, interactions, and/or message transports within a 
device environment to address these problems. 

SUMMARY 

Embodiments of the present invention address the prob- 
lems discussed above through application of an aggregator 
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to the device environment. An aggregator embodiment may 
be employed to act as a single point of communication 
between many devices thereby limiting the communication 
paths to the number of devices in the environment. An 

5 aggregator embodiment may also be employed to accept 
communications from multiple transport types to act as a 
translator between devices of different transports. An aggre- 
gator embodiment may also be employed to control device 
interaction by employing interaction rules. 

10 An aggregator device embodiment includes at least one 
transmitter and at least one receiver. The embodiment also 
includes a memory that stores interaction rules for the 
plurality of devices of the environment. A processor in 
communication with the transmitter, receiver, and memory 
is also included. The processor is configured to receive a 

15 message from a first device through the at least one receiver, 
reference the interaction rules in relation to the message 
from the first device to direct communications through the at 
least one transmitter to one or more devices of the plurality. 
The logical operations of the processor that involve 

20 implementing the interaction rules are embodied in methods. 
The methods specify how a particular device or group of 
devices communicate. One embodiment of a method 
involves transmitting a message to the aggregator from a 
first device and receiving the message at the aggregator. The 

25 aggregator references interaction rules in relation to the 
message from the first device to direct communications from 
the aggregator to one or more devices of the plurality. 

Another embodiment of a method involves detecting a 
change of state at a first device and in response to detecting 

30 the change of state, transmitting a change of state message 
to the aggregator. The aggregator receives the change of 
state message and references interaction rules in relation to 
the change of state of the first device to determine a change 
of state instruction for a second device of the plurality. The 

35 aggregator then transmits the change of state instruction to 
the second device. 

The various aspects of the present invention may be more 
clearly understood and appreciated from a review of the 
following detailed description of the disclosed embodiments 

4Q and by reference to the drawings and claims. 

DESCRIPTION OF THE DRAWINGS 
FIG. 1 is a diagram of a device environment. 
FIG. 2 is a diagram showing the major components of an 
embodiment of an interactive device. 
45 FIG. 3 is an exemplary operational flow of an interactive 
device communicating with other devices of the environ- 
ment. 

FIG. 4 is an exemplary operational flow of communica- 
tion between interactive devices involving a message broad- 
cast to all devices of the environment. 

FIG. 5 is an exemplary operational flow of communica- 
tion between interactive devices involving a message 
directed to a specific device of the environment. 
55 FIG. 6 is an exemplary operational flow of communica- 
tion between interactive devices involving a request and a 
response to the request. 

FIG. 7 is an exemplary operational flow of rule acquisi- 
tion of an interactive device. 
60 FIG. 8 is an exemplary operational flow of rule acquisi- 
tion of an interactive device involving learning by a device 
receiving a state change after a state change of another 
device. 

FIG. 9 is an exemplary operational flow of rule acquisi- 
65 tion of an interactive device involving learning by a device 
receiving a state change before a state change of another 
device. 
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HG. 10 is an exemplary operational flow of rule acqui- A device environment is shown in FIG. 1 and is repre- 
ssion of an interactive device involving a request for rules sentative of a small area such as within a single household 
to a device and a subsequent response. However, a device environment may expand beyond a single 

FIG. 11 is an exemplary operational flow of content area through networking of devices among various areas, 
control among interactive devices. 5 This simplified device environment 100 shows three devices 

FIG. 12 is an exemplary operational flow of media rights f ° f exe t m ^ P UI P oses > but any number of devices may be 

sharing among interactive devices present a given environment 100. The devices of the 

vttc ~t i ' j • ' . environment 100 are devices that customarily appear within 
•ii 2\ * a i agram u a deV1Ce environment that the particular type of environment. For example, in a house- 
illustrates the complexity that occurs in relation to interac- hold the devices would include but not be limited to typical 
tivity among an increasing number of devices. household devices such as a television, VCR, DVD, stereo, 

FIG. 14 is a diagram of a device environment including an toaster, microwave oven, stove, oven, washing machine' 

embodiment of an aggregator that also illustrates the major dryer, and telephone. These devices are adapted to become 

components of the aggregator. interactive as is discussed below. 

FIG. 15 is a diagram of an embodiment of an aggregator 15 Each device communicates with the other devices of the 
illustrating the components for translating among multiple environment 100 in this example. A first device 102 corn- 
communication transports. municates with a second device 104 through a bi-directional 

FIG. 16 is an exemplary operational flow of device communication path 108. The first device 102 communi- 

interaction involving an aggregator. cates with a third device 106 through a bi-directional com- 

FIG. 17 is an exemplary operational flow of device ™ munication path 110, and the second device 104 com muni - 

interaction involving an aggregator that learns interaction CateS ^ e third device 106 through a bi-directional 

rules and translates among multiple communication trans- communication path 112. The communication paths may be 

ports. wired, wireless, or optical connections and may utilize any 

FIG. 18 is a diagram of a device environment interacting ° f °? ^ eU " known P h y sicaI transmission methods for com- 

with notification devices interfaced with a user 25 mumcatm S devices in a relatively small relationship 

TTTo m- i , „ to one another. 

WO. 19 is an exemplary operational flow of interaction tu • , 

from a device environment to a remote notification device ma w ^ mmunlcatl0n ^ ™* between two devices 

through a remote communication transport. ^ & Up a c ™^°° transport. For example, two 

cir- m • ■ i • ... devices may utilize the Bluetooth transport, standard infra- 

from a ^^J^^^^ 101 ?^ ° f mteraCti0n 30 red tranS P° rt Where Une of ^ * mL^ } a "or 
*Zah ™ ° 0tlficaUon de f Vlce *> a de ™* environment VHF transport, and/or many others. Networked areas form- 
mrough a remote commumcauon transport. mg m environmenl can ^ ^ technology such * 

HG. 21 is a diagram of an embodiment of a device for Ethernet, WAN technology such as frame relay and the 

providing a display of information about a device environ- Internet. Multiple transports may be present in any single 

ment 35 environment. As discussed below with reference to FIGS. 15 

FIG. 22 is an exemplary screenshot of the device of FIG. and 17, a particular device such as an aggregator may be 

21 that illustrates a device menu and a learn mode menu. equipped to translate messages from one communication 

FIG. 23 is an exemplary screenshot of the device of FIG. transport to another. Aggregators are discussed generally 

21 that illustrates a learn mode allowing the user to select ^ * n more detail below. 

function representations on the screen to associate functions 40 The details of the devices 102, 104, and 106 are shown in 

of devices. more detail in FIG. 2. An interactive device 200 includes a 

FIG. 24 is an exemplary screenshot of the device of FIG. processor 206 that communicates with various resources 

21 that illustrates a learn mode allowing the user to select through a data bus 214. The processor 206 may execute 

functions on devices that are to be associated. software stored in a memory 208 or may utilize hardwired 

FIG. 25 is an exemplary screenshot of the device of FIG 45 di S ita . 1 lo S ic io perform logical operations discussed below 

21 that illustrates a rule display mode for visually conveying to bring about ^ device interaction. The processor 206 

the stored rules to a user. communicates with the memory 208 to apply interaction 

FIG. 26 is an exemplary operational flow of a learn mode mles A that u S overn ?» communications. Interaction rules 

where the user selects functions on the devices that are to be SpeC1 ^ , f P artlclllar communication should occur, the 

associated. 50 reci P ie nts of the communication, and the information to be 

T7w n >-%m • ! , n conveyed through the communication. Memorv 208 mav 

J^l:^™^ ° Peratl0nal fl ° W ° f 4 ^ delude electron! storage such as RAM and ROM, LS 

cio is • , magnetic or optical storage as well. 

HO . 28 is an exemplary operational flow of a learn mode The processor 206 communicates with a transmitter 212 

si n to fimcUon f re P resentatlons °» a 55 and a receiver 210 to physically communicate with the 0 £r 

screen to associate functions of devices. devices of the envir ^ ent . ^ transmitter mi receiver 

DETAILED DESCRIPTION pairs discussed herein for the various embodiments may be 

T , , . ' separate or incorporated as a transceiver. When an interac- 

Interaction among devices of an environment permit the tion rule specifies that a communication from device 200 

devices to perform automatic changes of state without 60 should occur, the processor 206 controls the transmitter 212 

requiring the user to individually control each device. to cause it to send a message. The message may take various 

Through recognition of patterns of user behavior, interactive forms discussed below depending upon the intended recipi- 

devices can associate the various user driven events from ents. The receiver 210 receives messages directed to the 

one device to the next to effectively create interaction rules. device 200. The communications among devices may be 

Application of these interaction rules allow the devices to 65 configured so that each device to receive a message has 

implement the state changes automatically through commu- identification data included in the message so that the 

nication of events between the devices. processor 206 determines whether a message is relevant to 
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the device 200 based on whether particular identification 
data is present. 

Alternatively, other schemes may be used to communicate 
wherein a physical parameter of the receiver 210 controls 
whether a device 200 receives the message as one intended 
for it to be received. Examples of such physical parameters 
include the particular frequency at which a signal is 
transmitted, a particular time slot during which the message 
is transmitted, or the particular type of communication 
transport being used. The transmitter and receiver may be of 
various forms such as a modem, an Ethernet network card, 
a wireless transmitter and receiver, and/or any combination 
of the various forms. 

The processor 206 also interacts with the intended func- 
tionality of the device 200. The device 200 includes com- 
ponents 202 that provide the unique function of the device 
200. If the device 200 is a television 224, then the compo- 
nents 202 include the circuitry necessary to provide the 
television function. One skilled in the art will recognize that 
the processor 206 can be separate and distinct from the 
processing capabilities of the components 202 or 
alternatively, may be wholly or in-part incorporated into the 
processing capabilities of the components 202. The compo- 
nents 202 of many devices have digital logic such as an 
on-board processor of a television 224, CD player 216, 
stereo system 218, dryer 220, or telephone 222. 

The processor 206 can control the operations of the 
components to cause state changes of the device 200. For 
example, the processor 206 can cause the channel to change 
on the television or cause the oven to preheat to a particular 
temperature. Thus, the processor 206 can reference interac- 
tion rules stored in memory 208 in relation to communica- 
tions received through receiver 210 to determine whether a 
state change is necessary or can receive state change instruc- 
tions through receiver 210 and implement the requested state 
change. 

Additionally, the device 200 includes a sensor 204 for 
providing state change information to the processor 206 
about the device 200. The sensor 204 may be either a 
physical sensor such as a transducer for detecting motion or 
a thermocouple for detecting temperature, or the sensor 204 
may be a logical sensor. The logical sensor may be a 
programmed function of processor 206 or the processor of 
components 202 or may be hardwired logic. A logical sensor 
may be separate and distinct from the processor 206 and/or 
the digital logic of components 202 and communicate 
through the bus 214, or it may be incorporated wholly or in 
part in either the processor 206 of the processor of the 
components 202. The logical sensor 204 acts as an interface 
to the digital logic of the components 202 for detecting the 
logical state of a component, such as a particular input that 
is active on a stereo, or a particular channel being displayed 
on a television. 

The processor 206 receives input from the sensor 204 to 
determine a current state of the components 202 and thereby 
determine when a change of state of the device 200 occurs. 
As discussed below, changes of state are used to learn 
interaction rules and implement the rules once they have 
been learned. Implementing interaction rules involves con- 
trolling changes of state at device 200 and/or transmitting 
change of state information about device 200 to other 
devices or transmitting change of state instructions to other 
devices, 

FIG. 3 shows the basic operational flow of the processor 
206 for implementing device interaction to send a commu- 
nication from device 200. A state change is detected at the 
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device 200 as described above at detect operation 302 by the 
sensor 204. The state change may be a user driven event, 
such as a user turning the power on for the television, or an 
automatically occurring event such as an oven reaching a 

5 preheat temperature. 

After detecting the change of state, the processor 206 
references the rules of device interaction stored in the 
memory 208 to determine whether a communication is 
necessary, who should receive the communication, and the 

10 particular information to include at rule operation 304. The 
processor 206 performs a look-up of the state change that 
has been detected to find the interaction rule that is appro- 
priate. The processor 206 then communicates according to 
the appropriate interaction rule by sending a message 

15 through transmitter 212 at communicate operation 306. 

FIG. 4 shows an operational flow of a specific type of 
communication where a device 200 publishes its state 
change to all devices via a broadcast so that all devices 
receive the message. A broadcast to all devices is useful 

20 when devices are attempting to learn interaction rules by 
observing state change events occurring within the device 
environment 100 during a small interval of time. 

The operational flow begins at detect operation 402 where 
the processor 206 realizes that sensor 204 has detected a 
change of state at device 200. The processor 206 then 
determines that a broadcast is appropriate at determine 
operation 404. The processor 206 may make this determi- 
nation by referencing the rules of interaction to determine 

3Q whether a broadcast is indicated. If learn modes are provided 
for the devices, as discussed below, then the processor 206 
may recognize that it is operating within a learn mode where 
broadcasts of state change are required. 

Once it is determined that a broadcast to all devices is 

35 appropriate, the processor 206 causes the broadcast to occur 
by triggering the transmitter 212 to send the message to all 
devices of the environment at broadcast operation 406. As 
discussed above, messages may be addressed to specific 
devices by manipulation of a transmission frequency, a time 

40 slot of the transmission, or by including recipient identifi- 
cation data in the transmission. The message contains an 
identification of the device 200 and the particular state 
change that has been detected. 
The devices of the environment receive the message at 

45 receive operation 408. In this exemplary embodiment 
shown, the devices make a determination as to whether a 
reply is necessary at determine operation 410. Such a 
determination may be made by the devices by referencing 
their own interaction rules or determining that a learn mode 

50 is being implemented and a reply is necessary because they 
have also detected their own state change recently. When a 
reply is necessary, the one or more devices of the environ- 
ment send a reply message addressed to the device 200 at 
send operation 412, and the device 200 receives the message 

55 through receiver 210 at receive operation 414. 

FIG. 5 shows an operational flow where a message is 
directed to a specific device of the environment from the 
device 200. The processor 206 recognizes that the sensor 
204 has detected a change of state of the device 200 at detect 

60 operation 502. The processor 206 then determines from the 
interaction rules that a second device is associated with the 
state change at determine operation 504. The second device 
may be a subscriber, which is a device that has noticed 
through a learning operation that it is related to the first 

65 device 200 through a particular state change event and that 
the first device should provide it an indication when the 
particular state change event occurs. Once it has been 
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determined who should receive a message, the processor first device determines that a broadcast is appropriate and 

206 triggers the transmitter 212 to direct a message to the sends the broadcast of the state change to all devices of the 

second device at send operation 506, and the message environment at broadcast operation 804. All devices receive 

includes a notification of the state change of the first device the broadcast at receive operation 806. 

- , 5 Aft** receiving the broadcast, each device of the envi- 

The processor 206 may employ additional logic when ronment monitors for its own state change. A second device 

directing the message to the second device. The processor that received the broadcast detects its state change at detect 

u r^.u fr ° m the lnteracUon ™les in memory 208 operation 808 within a predetermined period of time from 

whether the second device should change state in response when the broadcast was received. The second device then 

to the detected change of state of the first device 200 at query io creates the interaction rule by associating the state change of 

operation 508. If so, then the processor 206 includes an the first device with the state change of fhe second device at 

instruction ,i i the message to the second device at message rule operation 810. The rule is stored in the memory of the 

operaUon 510 that specifies the change of state that should second device so that the processor can apply the rule 

be automatically performed by the second device. thereafter. 

FIG. 6 shows an operational flow where a request is made 15 At application operation 812, the second device receives 

and a response is thereafter provided. At detect operation the state change message from the first device and then 

SS, ^T^ 1 ff , reC0 ^ ^ ?l SenS ° r 204 haS a PP Ues ^ interaction rule that has been learned to auto- 

a ^ 1 ' ? 6V1Ce 20 °- ^ prOCeSSOr maticaU y chan S e its state accordingly. The second device 

206 tiien determines that a second device is associated with applies the interaction rule by looking up the state change of 

toe state of change at determine operation 604. In this case, 20 the first device in its memory to see if there is an association 

die processor 206 recognizes that a request to the second with any state changes of the second device. The previously 

device is necessary, such as by reference to the interaction learned rule specifies the state change of the second device 

rules or due to some other reason such as a particular learn and the second device automatically changes state withou 

mode being implemented. requiring ^ ^ t0 mamia]]y mpegt ^ change 

The processor 206 triggers the transmitter 212 to direct a 25 As an example of this method of learning, the user may 
request message to the second device at send operation 606. turn on the VCR which sends a broadcast of the state change 
The request message can specify that the second device is to The user shortly thereafter tunes the TV to channel 3 to 
respond by transmitting particular data that the second watch the VCR signal. The TV has received the broadcast 
device oirreatly possesses in memory to the device 200. The from the VCR prior to the user tuning to channel 3 and 
f eC °<n. receiv f s J the reqUCSt meSS3ge at receive °P era - ^erefore, *» ^ associates the tuning to channel 3 with the 
tion 608. The second device prepares a response by obtain- VCR being powered on to learn the interaction rule 
mg the required information from its memory, sensor, or Thereafter, when the user turns on the VCR the TV auto- 
components. Such information includes interaction rules, its matically tunes to channel 3. 
current state, its current capabilities, or those who have err d,i„,™™.i. .l j r, 
subscribed to it for state change events Once the informa 35 fi ? ^ ^alive method «* Naming where the 
tion is obtained, the second'devS sends «? esZse ™T e 'Sf? * ^ * ^ l ?™ **> interacti ° n 
including the information to the first device 20oTsTnd * ' tStSTSSL T * T ° P ?^ T 
operation 612 wnere a nrst device detects its own change of state. In 

CTr _. ' , a . , . response to the change of state, the first device then begins 

devJS 200 ™ °, a ammg Pr0CeS 1 ° f thC <° moni,orin g for incoming state change messages at monitor 

device 200. The device 200 may learn interaction rules that operation 904. Subsequently, a second device receives a 

it obeys by observing activity in the environment in relation change of state at detect operation 906 and broadcasts the 

to its own state changes Because the device may automati- change of state message to all devices at broadcast operation 

cally learn interaction rules rather than requiring that they be 908. The broadcast is effectively a request that any device 

manually programmed, a burden on the user is lessened. The 45 previously experiencing a state change add the second 

operational flow begins by observing the environment to device to its subscriber list 

detect a state change message at observation operation 702. Whi i e monitoring; the fiist device receives ^ ch f 

The state change message may originate from another *t*,„ m „ e. .u " ! receives me cnange or 

AA . . .. , , ^ " own change of state, the first device creates an interaction 

Alter detecting state change messages, the processor 206 rule at rule operation 912. The first device creates the 

learns the rale at learn operation 704 by associating together interaction rule by adding the second device and state 

state changes that have occurred over a small interval of change to its subscriber list that is associated with its state 
h . me /°' exam P le ' ause r may turn on one device and then 55 change. Subsequently, the first device detects its state 

shortly thereafter manipulate another device, and these two change at detect operation 914 and then directs a message to 

state changes are observed and associated as a rule. Partial- the second device at message operation 916 in accordance 

Iar methods of learning are discussed m more detail below with the interaction rule learned by the first device 

„ , , . , . call y performs the appropriate state change with no input 

FIG. 8 shows the logical operations where the device from the user at state operation 918. As an example of this 

whose state changes later in time leams the interaction rule. 65 method of learning, the user turns on the VCR which begins 

The operations begm at detect operation 802 where a first to monitor for a state change broadcast. The user tunes the 

device detects a change of state through its state sensor. The television to channel 3 shortly thereafter, and the television 
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broadcasts the state change . The VCR receives the broadcast example, the first device for media playback may be portable 

and associates the TV to channel 3 state change with its and may easily be taken to other device environments that 

power on state change . After the rule is created and when the impose more stringent restrictions on media content than the 

user powers on the VCR, the VCR executes the rule by first device imposes on itself. At query operation 1107, the 

sending a message with instruction to the TV. The TV 5 first devices detects whether it is configured to obey content 

implements the instruction to automatically tune to channel *iles of the environment in addition to its own content rules. 

3 If the first device is configured to obey only its own content 

' , .,. 11T . rules, then the first device begins normal playback of the 

FIG. 10 shows another alternative learning method. For media at play5ack opera tion 1108. The first device may 

this method, it is assumed that a device already has one or reference content rules at this point at settings operation 

more interaction rules. The logical operations begin at send ™ mo to determine whether the content being played back has 

operation 1002 where a first device sends a request to a m assoc iated preferred setting or setting limitation. For 

second device. The request is for the interaction rules stored example, a user may have configured a rule that a particular 

by the second device. The rules of the second device may be mov ie is to be played back at a preferred volume setting or 

relevant to the first device for various reasons such as mat me volume for playback cannot exceed a particular 

because the first device is involved in the interaction rules of * 5 se tting. The first device implements the preferred setting or 

the second device or because the first device is acting as an limitation during playback. 

aggregator that controls interaction of the environment. The M ^ &st device fc configured to obey its own content 

details of the aggregator are discussed in more detail below mles as well ^ tGe content mles of my environment where 

After the first device has sent the request, the second it is placed, then after determining that the media content is 

device receives the request at receive operation 1004. The playable according to its own rules, operational flow tran- 

second device then retrieves its interaction rules from sitions from query operation 1107 to send operation 1114. 

memory at rule operation 1006. The second device then Additionally, if query operation 1104 detects that the first 

sends a reply message to the first device at send operation device does not have an applicable content rule, then opera- 

1008 that includes the interaction rules of the second device. tional flow transitions directly to send operation 1114. 
The first device receives the reply message with the rules at 25 At send operation 1114, the first device transmits a 

receive operation 1010 and stores the rules in memory at rule message having the content information previously obtained 

operation 1012. Thereafter, the first device can apply the to a sec0 nd device that maintains content rules for the 

rules stored in memory to control state changes upon user current environment where the first device is located. The 

driven events at application operation 1014. ^ second device receives the message with the content infor- 

Device interaction also permits additional functionality mation and compares the content information to the stored 
among the devices of an environment such as the control of content rules at rule operation 1116. The comparison to the 
media content to be played within the environment and the content rules again involves content ratings and/or rights, 
control of device settings dependent upon the particular settings, and/or setting limitations. The details of this corn- 
media being played. FIG. 11 shows the logical operations of 35 parison are also shown in FIG. 11. 

device interaction involving a device, such as an aggregator, The comparison begins at query operation 1126 where the 

that is in charge of the content control and/or content settings second device detects whether the content is playable in 

for a device environment. For example, a user may set up a relation to the content rules. The content rules may specify 

parental control at one device, and the one device then a maximum rating and/or whether digital rights exist for the 

becomes the instigator of content control for other media 4Q content being played. Other limitations may also be speci- 

playback devices of the environment. Also, where digital fi e d in the content rules for comparison to the content 

rights are required for playback, the instigator of content information, such as a limitation on adult language present 

control may manage those digital rights to prevent unautho- ^ lne content that is indicated by the content information. If 

rized playback. the content is not playable, then the comparison indicates 

The logical operations begin at content operation 1102 45 that a stop instruction should be sent at stop operation 1130. 

where a first device is attempting to play media. Here, the If the content is playable, then the comparison indicates that 

first device obtains content information included within the a play instruction should be sent along with any associated 

media, such as recognizing the title of a CD or DVD that is settings or setting limitations at playback operation 1128. 
about to be played. Obtaining the content information Once the comparison is complete, the second device 

applies for devices that support multiple media formats, 50 directs a message to the first device at send operation 1118, 

such as a DVD player obtaining content information from and the message instructs the first device according to the 

DVDs or audio CDs during playback. Then, at query opera- stop instruction or playback instruction resulting from the 

tion 1104, the first device detects whether it has its own previous comparison to the content rules. The first device 

content rules. If so, then the first device detects whether the receives the message and interprets the instruction at receive 

content is playable by comparing the content information to 55 operation 1120. The first device then implements the 

the associated content rules. At least two checks may be received instruction to either stop playing the media content 

done at this point, one for content ratings and one for content or begin playback with any specified settings or limitations 

rights. Content ratings are limits on the ratings of media that at implementation operation 1122. 

can be played, such as no content worse than a PG rated As an option, the first device may then create a content 

movie or no content with a particular type such as excessive 60 m i e that associates the instruction with the content infor- 

adult language. Content rights are digital rights for playback mation at rule operation 1124 if the first device does not 

authorization that prevent copyright or license infringement. already have a local content rule. By creating the rule at the 

If the content is not playable, then the first device stops first device, the first device will at query operation 1104 

playback at stop operation 1112. If the content is playable, detect that a content rule exists on subsequent attempts to 

then two options may occur depending upon whether the 65 play the same content. The first device will then handle its 

first device is configured to obey content rules from a device own content control without requiring communication with 

environment in addition to its own content rules. For the second device. 
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The second device may obtain content rules through paths 1320, 1328, 1334, 1338, and 1342; and a sixth device 

various methods. For example, the second device may 1312 uses paths 1322, 1330, 1336, 1340, and 1342). 

receive a message from a third device at receive operation The complexity in coordinating the communications and 

1132, and the message specifies a content rule. A user may interactions in such a crowded environment 1300 may result 

have selected a content rule at the third device for media 5 m inadequate bandwidth for the communication channels, 

playback, and the third device then provides the rule to the cross-talk between the channels, and incompatible transports 

second device as an automatic function or in response to a between devices. Furthermore, unintended rules may be 

request for content rules from the second device. The second learned because one or more of the devices may be unrelated 

device creates the content rule by storing it in memory at t0 tne ot h ers . F or example, one person may answer the 

rule operation 1134. 10 telephone shortly before another person starts the clothing 

During playback, the first device periodically repeats the dryer. There was no intended relationship but the phone or 

comparison to environmental content rules at rule operation the dryer may associate the two state changes as an inter- 

1125, as was initially done at rule operation 1116. This action rule, which the users never intended, 

operation 1126 is done periodically because if the first An aggregator 1402 as shown in FIG. 14 may be intro- 

device is portable it may change locations after the start of ^ duced into a crowded environment 1400 to alleviate one or 

playback. In that case, if the playback was initially permis- more 0 f the concerns. As shown in FIG. 14, an aggregator 

sible but later becomes impermissible because the first 1402 can be used to reduce the number of bi-directional 

device enters a more restrictive device environment, then communications paths. For the six device environment, the 

playback stops as indicated at stop operation 1130. aggregator has reduced the number of paths down to six 

FIG. 12 shows logical operations demonstrating the bor- 20 (device 1414 uses path 1426, device 1416 uses path 1428, 

rowing of media rights of a content rule from a device, such device 1418 uses path 1430, device 1420 uses path 1432, 

as an aggregator, that maintains the content rules. The device 1422 uses path 1434, and device 1424 uses path 

logical operations begin by a first device sending a request 1436). The aggregator 1402 acts as a conduit of communi- 

to borrow media rights to a second device that maintains the cation from one device to another, and may also be config- 

media rights at send operation 1202. For example, the first 25 ured to control or otherwise manage functions of a single 

device may be an MP3 player and the request is for device. 

permission to play a particular song or volume of songs. The aggregator 1402 uses a transmitter 1408 and receiver 

The second device receives the request and determines if 1406 capable of communicating with the multiple devices, 

the media rights to the content exist at receive operation 3Q The transmitter 1408 and receiver 1406 may be configured 

1204. If so, and they are not flagged as borrowed, then the to receive from all devices using various techniques known 

second device sends the media rights to the first device to in the art. For example, frequency division multiplexing, 

allow the first device to play the content at send operation time division multiplexing, code division multiplexing, opti- 

1206. The media rights are then flagged as borrowed at the cal multiplexing, and other multiplexing techniques may be 

second device at flag operation 1208. Subsequently, when a 35 used for a particular environment 1400 so that multiple 

third device requests authorization for media playback from devices can communicate with the aggregator 1402. 

the second device for the same content at send operation The aggregator 1402 also has a processor 1404 and a 

1210, the second device then checks the media rights at test memory 1410. The processor 1404 communicates with the 

operation 1212 and detects the flag. The second device then transmitter 1408, receiver 1406, and memory 1410 through 

sends a stop instruction to the third device at send operation 4Q a bus 1412. The aggregator 1402 may be incorporated into 

1214 to prevent the third device from playing the content a particular device of the environment as well so that the 

because the first device already has rights to it. aggregator includes the additional device features such as 

These logical operations could also be adapted to provide components and a state sensor discussed in relation to FIG. 

a count for the media rights so that more than one device can 2. The logical operations of an aggregator such as shown in 

access the media rights for playback of content if multiple 45 FIG. 14 are discussed below. 

rights are owned for the content. Each time the media rights The processor 1404 of the aggregator may be configured 

are borrowed by a device, the count of media rights is to perform various advanced functions for the device envi- 

decremented. Once the count reaches zero, stop instructions ronment. The processor 1404 may be configured to perform 

are sent to the devices subsequently attempting playback. periodic review of interaction rules to edit rules that are 

Furthermore, there can be a similar device exchange to 50 inappropriate for various reasons. For example, memory 

unflag the digital rights or increment the count to restore 1410 may contain a list of impermissible associations that 

capability of other devices to subsequently borrow the the processor 1404 may refer to when reviewing interaction 

digital rights to media content. ' rules. If an impermissible association is found the, commu- 

The device interactions discussed above including general nication link that causes the problem may be excised from 

interaction to bring about states changes, interactive learning 55 the rule. Additionally, the processor 1404 may be configured 

of interaction rules, and interactive content control become to entirely remove interaction rules that are inappropriate, 

increasingly complicated as the number of devices in the The processor 1404 may also be configured to support 

environment increase. As shown in FIG. 13, when the complex interaction rules. For example, devices may be 

number of devices of an environment 1300 grows to six, the separated into classes so that actions of one device may only 

number of bi-directional communication paths grows to 60 affect devices within the same class. The processor 1404 

fifteen to ensure that every device can communicate directly may reference such class rules in memory 1410 to filter out 

with every other device. Each device uses five bi-directional faulty rules that might otherwise be learned, such as those 

paths (a first device 1302 uses paths 1314, 1316, 1318, 1320, where devices of different classes are involved, 

and 1322; a second device 1304 uses paths 1314, 1324, Furthermore, the processor 1404 may be configured to 

1326, 1328, and 1330; a third device 1306 uses paths 1316, 65 develop rules based on conditional logic or iterative logic, 

1324, 1332, 1334, and 1336; a fourth device 1308 uses paths and perform multiple activities of a rule in series or in 

1318, 1326, 1332, 1338, and 1340; a fifth device 1310 uses parallel. 
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As an example of conditional logic being employed, a rule 
may specify that a phone ringing means the volume should 
be decreased for several different devices but only if they are 
actively playing content. Then when the phone hangs up, the 
volume should be increased but only for those devices 
whose volume was decreased by the phone ringing. An 
example of iterative logic provides that the front porch lights 
should be turned on at 6 p.m. and off at 12 a.m. everyday. 

An example of serial execution of interaction rules with 
multiple activities provides that when a light on a computer 
desk is turned on, the computer is then turned on, and after 
that the monitor is turned on followed by the computer 
speakers being turned on. An example of parallel execution 
of interaction rules with multiple activities provides that 
when a person is exiting a room, all devices of the room are 
powered off simultaneously. 

FIG. 15 illustrates an embodiment of an aggregator 1502 
that is additionally configured to translate among different 
communication transports of the device environment 1500. 
Device 1518 may communicate through signals 1524 of a 
first communication transport while device 1520 communi- 
cates through signals 1522 of a second communication 
transport. For example, the first communication transport 
may be Ethernet while the second communication transport 
is fiber optical. The communication transports may differ in 
the physical mode of transferring signals (e.g., Ethernet 
versus fiber optical) and/or in the logical mode (a first data 
encoding scheme versus a second). 

The aggregator 1502 includes a first transmitter 1508 and 
receiver 1506, separate or combined as a transceiver, for 
communicating across the first communication transport. 
The aggregator may also include a second transmitter 1512 
and receiver 1510, separate or combined as a transceiver, for 
communicating across the second communication transport 
where the second communication transport differs in the 
physical mode of transport. Aprocessor 1504 communicates 
with memory 1514 and the two transmitter-receiver pairs 
through a bus 1516. Although two transmitter-receiver pairs 
are shown for two communication transports, one skilled in 
the art will recognize that any number of transmitter- 
receiver pairs and communication transports may be 
utilized, including only one, depending upon the different 
number of physical transports to support within the device 
environment. 

The processor 1504 detects from the messages being 
received where communications should be directed. This 
includes determining whether the messages should be trans- 
lated to a new communication transport when sending the 
message to the intended device. The processor 1504 may 
perform the same logical operations of the processor of 
aggregator 1402 with the addition of translation operations 
from one transport to another where necessary. 

FIG. 16 shows the basic logical operations of an aggre- 
gator. The logical operations begin when a first device 
transmits a message to the aggregator at send operation 
1602. The first device may send a message to the aggregator 
that is intended as a broadcast to all devices, as a message 
directed to a specific device, or as a message intended solely 
for the aggregator. The aggregator receives the message at 
receive operation 1604. 

The aggregator then references interaction rules that it 
maintains in memory in relation to the message it has 
received at rule operation 1606. For example, the environ- 
ment may be configured so that the devices maintain no 
interaction rules other than to direct a message for every 
state change to the aggregator and rely solely on the inter- 
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action rules of the aggregator to bring about subsequent 
activity in the environment. The environment may alterna- 
tively be configured where the devices maintain interaction 
rules and provide instruction to the aggregator with each 

5 message, so that the aggregator acts upon the instruction to 
bring about subsequent activity. 

After the aggregator has received the message and 
referred to the interaction rules in relation to the message, 
the aggregator communicates with devices of the environ- 

10 ment in accordance with the interaction rules and any 
received instruction from the first device at communication 
operation 1608. For example, the aggregator may possess 
the interaction mle that when the VCR is on, the TV should 
be tuned to channel 3. When the aggregator receives a power 

15 on message from the VCR, the aggregator then sends an 
instruction to the TV to tune to channel 3. Alternatively, the 
power on message may instruct the aggregator to send an 
instruction to the TV to tune in channel 3. 

FIG. 17 shows the logical operations of an embodiment of 

20 an aggregator, such as the aggregator 1502 of FIG. 15. The 
logical operations begin at detect operation 1702 where the 
first device detects its own state change. The first device then 
sends a message to the aggregator at send operation 1704. 
The aggregator receives the message at receiver operation 

25 1706 and references its interaction rules at rule operation 
1708 in relation to the received message indicating the state 
change. 

The aggregator tests whether to switch communication 

30 transports at query operation 1710 by referencing its inter- 
action rules. The interaction rules specify how to commu- 
nicate with each device. The aggregator learns the one or 
more devices to communicate with in response to the 
message from the first device by either looking up the state 

35 change of the first device in the interaction rules to ..find 
associations or by interpreting an instruction from the" first 
device included in the message. After determining the 
proper device to communicate with, the aggregator can look 
up the device in memory to determine which communication 

40 transport to employ. 

Once the aggregator has determined which transport to 
use for the next communication, the message from the first 
device or a new message from the aggregator is prepared by 
translating to the second communication transport appropri- 

45 ate for the next communication at translate operation 1712. 
Where only the logical mode of communication transport 
differs, a second communication transport may not be 
needed. Furthermore, the aggregator may act as a conduit 
where no change in the physical or logical mode of transport 

5 0 should occur. As an example of where a change in transport 
does occur, the aggregator may receive a message from the 
VCR via infrared airwave signals and then prepare a mes- 
sage to the TV to be sent via a fiber optical connection. The 
aggregator sends the message to the second device at send 

55 operation 1714. The second message may instruct the sec- 
ond device that the first device has changed state if the 
second device has its own interaction rules, or the message 
may provide a specific instruction to the second device. 
After receiving the message, the second device imple- 

60 ments any instruction or automatic state change dictated by 
its own interaction rules. The second device may respond to 
the aggregator if necessary at send operation 1716. The 
return message may be an indication to the aggregator of the 
state change that the second device has performed or may be 

65 a reply to a request from the aggregator such as for current 
state, capabilities, or rules. The aggregator again references 
its interaction rules at rule operation 1718 to determine the 
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next action after receiving the message from the second communicating to both devices defined by proximity and 
device The aggregator then communicates with other external devices is present in the environment 1800. The 
devices of the environment as necessary at communicate device 1802 includes at least one transmitter 1814 and 
operation 1720 receiver 1812 for communicating with proximity based 
P , . * £ - . c devices 1818 in the environment 1800 over a communica- 
te logical operaUons for the aggregator learning fee don oft mQ ^ deyice lg02 0 f includes a memory 
interaction rules being applied are also shown in FIG. 17. lg06 ^ stQres interaction ^ md a pr0 cessor 1804 for 
Several possibilities exist for learning rules at the aggrega- executing the functions of the device 1802. The processor 
tor. Auser interface discussed below may be provided so that 1804 communicates through the bus 1816. The memory 
a user enters interaction rules at user operation 1722. The mav store translation rules in the embodiment 
aggregator may observe closely occurring state change 10 wnere communication with notification devices is sup- 
broadcasts that are associated as interaction rules at obser- ported. 

vation operation 1724, as was discussed above for learning ^ device lg02 of m js embodiment also includes at least 

with individual devices. The aggregator may request that a Qne tran smitter 1810 and receiver 1808 that communicate 

particular device forward its interaction rules to the aggre- through a remote communications transport 1828 to external 

gator where they can be stored and implemented at request ^ devices. The remote communications transport 1828 may 

operation 1726 . take various forms such as a conventional telephone network 

After receiving the interaction rule in one of the various 1822 including a central office 1826. The remote commu- 

ways, the aggregator stores the interaction rule at rule nications medium may additionally or alternatively involve 

operation 1728. When state change messages are received at ^ a wireless network 1824 for mobile telephones or for pagers, 

the aggregator and the aggregator references the interaction Communication can be established between the device 

rules such as at rule operation 1708, the aggregator com- jgQ2 m( [ a remotely located telephone 1830, computer 

pares information in the message to the stored rules at 1832, or wireless communication device 1834 such as a 

comparison operation 1730. Through the comparison, the mobile phone or pager which is explicitly defined in 

aggregator determines the appropriate action to take to ^ memory 1806 as being part of the device environment. The 

complete communication to other devices. device 1802 can relay information between itself or other 

Device interaction within the device environment allows proximity based devices of the environment 1800 and the 
the burden on the user to be lessened while the user is remotely located communication devices, 
present within or absent from the environment. However, j n the embodiment where notification devices are 
under certain scenarios the user is absent but needs to remain 3Q supported, the user can remain in contact with the device 
in contact with the device environment. For example, the environment 1800 by communicating through the notifica- 
user may need to know when the oven is finished cooking so tion devices that are either external, such as devices 
the user can return home, or the user may need to delay the 1830-1834, or are proximity based, such as device 1836. For 
oven from automatically preheating at a certain time because example, the device 1802 may send short messages to a 
the user will be late. Therefore, for these scenarios the 35 mobile phone 1834 or to a proximity based portable corn- 
device environment needs to communicate remotely with munication device 1836 if the user is in proximity. The 
the user. device 1802 may provide machine speech or text that can be 

FIG. 18 shows one illustrative case of device communi- interpreted by the user as a notification of a state of the 
cation where the messages extend beyond a closely defined environment. Similarly, the user may send machine tones, 
area, such as a single room or household, to an external or 40 speech, or text back to the device 1802 that can be inter- 
broader area. The external area includes any destination preted by the device 1802 as an instruction for the environ- 
reachable via a communications network. Thus, in this ment. 

illustrative case, the device environment is not defined by p or example, to implement the notification process the 

proximity but by explicit definition by the user. Such explicit processor 1804 may recognize a set of voice commands, 

definition may be provided by the user in many ways, such 45 tones, or text and translate those into instructions for various 

as through a listing stored in memory that describes the devices by referencing translation rules to interpret the 

devices and their address where they may be accessed instruction. The processor 1804 may then reference interac- 

through communication networks including the Internet, UO n rules to communicate the instruction to the appropriate 

wireless communication network, and landline telephone device based on identification received in the message from 

network. Thus, as used herein, device environment should 50 the remote device. Likewise, the processor 1804 may choose 

be understood to include both environments defined by from a set of machine voice commands, tones, or text to 

proximity as well as explicitly defined environments. communicate messages from the environment back to the 

Additionally, FIG. 18 shows an illustrative case of device user when the interaction rules indicate that the remote 

communication where notification messages are passed device should be contacted. 

between a notification device that is interfaced with the user 55 FIG. 19 shows the logical operations for communication 
and devices of the environment not otherwise interfaced from the device environment to the notification device 
with the same user. Thus, messages may be passed to the 1830-1834 or 1836. The logical operations begin at detect 
user from devices of the environment and from the user to operation 1902 where a first device of the environment 
the devices without the user interacting directly with those detects its own state change. The first device itself or a 
devices that send or receive the message. Such notification 60 dedicated device for remote communications such as an 
devices may be external, as discussed above, in that they are aggregator may then reference rules for interaction to deter- 
not part of the device environment through proximity but by mine whether a notification communication is necessary 
explicit definition by the user, or the notification devices based on the state change at rule operation 1904. For 
may be in close proximity and be included in the device example, if the previously discussed content control device 
environment on that basis. 65 detects that unacceptable content playback is being 
A device 1802 such as an aggregator for sending notifi- attempted, a notification may be provided to the notification 
cations to the notification device of the user and/or for device 1836 or 1830-1834. 
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The interaction rules may provide a hierarchy of commu- 
nication with notification devices, or for other non- 
notification devices as well, so that a particular state change 
may require that communications cycle through a list of 
devices until a response is received or the list is exhausted. 
At detect operation 1914, the appropriate device of the 
environment determines from the interaction rules the order 
of communication that should occur. For example, a par- 
ticular state change may require that a page be left with the 
user followed by a call to a mobile phone if there is no 
response to the page within a certain amount of time. 

The logical operations of FIG. 19 assume that the noti- 
fication device is an external device that is explicitly defined 
by the user. Thus, after determining the one or more noti- 
fication devices to contact, the device of the environment 
references translation rules at rule operation 1906 to deter- 
mine how to convey the message to the remotely located 
notification device that should be contacted. The translation 
rules are typically specified by the user directly at input 
operation 1916. Through a user interface, the user can 
specify the hierarchy and the particular translation rules to 
use. For example, the user can specify that a pager is 
contacted by dialing a specific telephone number over the 
ordinary telephone network, and that a text message should 
be left upon an answer. Rules may also include constraints 
such as the range of time when a particular notification 
device should be contacted. 

TTie device of the environment executes the interaction 
rule and translation rule to communicate remotely to a 
second device (i.e., a notification device) at communication 
operation 1908. As one exemplary option where a hierarchy 
is employed, the device tests whether the second device has 
responded at query operation 1910. If so, then the logical 
operations return to await the next state change requiring 
remote communications. If not, then the device of the 
environment communicates remotely to a third device (i.e., 
a different notification device) as specified in the hierarchy 
at communication operation 1912 again with reference to the 
interaction and translation rules. Cycling through the 
devices of the hierarchy continues until query operation 
1910 detects a response or the list is exhausted. 

Another exemplary option is to communicate with one or 
more notification devices without regard to a hierarchy. 
After the device of the environment has provided a com- 
munication to the second notification device, then a com- 
munication is automatically provided to the third notifica- 
tion device at communication operation 1912. This 
continues for as many remote devices as specified in the 
interaction rules. 

FIG. 20 shows the logical operations for communications 
from the notification device back to the device environment. 
At send operation 2002, the notification device directs a 
message to a first device of the environment that completes 
notification communications. For example, where the noti- 
fication device is external to the proximity defined device 
environment, the first device may maintain a connection to 
a telephone line, and the user dials the number for the line 
to contact the first device. The first device answers the call 
and awaits data signals from the remote notification device. 
The remote notification device then provides the message by 
the user speaking or using dialing tones. 

The first device receives the message at receive operation 
2004 and translates the message for transport to a second 
device of the environment at translate operation 2006. 

The first device may translate the message by referencing 
translation rules to convert the message into a form usable 
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by the second device and by referencing interaction rules to 
determine that the second device should be contacted. For 
example, according to the translation rules, an initial "1" 
tone from the remote device may indicate that the oven 

5 should be contacted, and a subsequent "2" tone from the 
remote device may indicate that the oven should cancel any 
automatic preheating for the day. 

Thus, translate operation 2006 involves determining the 
second device to communicate with through detecting an ID 

10 of the second device from the message of the remote device 
at ID operation 2020. In the example above, the ID of the 
oven is an initial "1" tone. The first device receives the "1" 
tone and references a "1" tone in the interaction rules to 
determine that a message should be sent to the oven. The 

15 first device receives the "2" tone and, knowing that the 
message is for the oven, references a "2" tone for the oven 
in the translation rules to determine that a cancel preheat 
message to the oven is necessary. The message is commu- 
nicated from the first device to the second device at com- 

20 munication operation 2008. 

The second device receives the message and implements 
the instruction at implementation operation 2010. For the 
example above, the oven receives a message instructing it to 
cancel its pre-programmed preheat operation for the day, and 

25 it cancels the preheat operation accordingly. As an exem- 
plary option to the logical operations, the second device may 
then send a message back to the first device confirming it has 
implemented the instruction at send operation 2012. 

3Q The first device 2014 receives the message from the 
second device at receive operation 2014, and then the first 
device 2014 translates the confirmation to a message that 
can be sent to the notification device at translate operation 
2016 in the instance where the notification device is exter- 

35 nal. For example, the first device 2014 may determine from 
the translation rules that it should send a pattern of tones to 
the telephone used to place the call to signal to the user that 
the oven canceled the preheat operation. The first device 
2014 then communicates the message to the remote notifi- 

4Q cation device over the remote communication transport at 
communication operation 2018 to complete the notification 
communications. 

The user may be provided a user interface to interact 
directly with devices of the environment such as the aggre- 

45 gator. As discussed above, the user may program interaction 
rules, translation rules, and hierarchies for remote commu- 
nication through the user interface. Additionally, the user 
may review information about the device environment 
through the user interface, such as current states of devices 

50 and existing interaction rules and translation rules of the 
environment. 

FIG. 21 shows the major components of an exemplary 
device 2102 establishing a user interface for the device 
environment. The user interface 2102 may be a separate 

55 device or may be incorporated into a device of the environ- 
ment such as an aggregator. The user interface 2102 includes 
a processor 2104 for implementing logical operations of the 
user interface. The processor 2104 communicates with a 
memory 2106 and a display adapter 2108 through a bus 

60 2110. The processor 2104 references the rules stored for the 
environment in the memory 2106 to provide information to 
the user on a display screen 2112 driven by the display 
adapter 2108. 

The user interface 2102 may provide several mechanisms 
65 for receiving user input. As shown, a touchscreen 2112 is 
provided so that the user can make selections and enter 
information by touching the screen 2112 that displays select - 
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able items such as text or icons. One skilled in the art will displaying the device representations at display operation 
recognize that other user input devices are equally suitable, 2802. A user selection of a first device selection is selected 
such as but not limited to a keyboard and mouse. at input operation 2804. The function representations of the 
Several exemplary screenshots of the user interface are first device are displayed on the screen for the first device at 
shown in FIGS. 22-25. The screenshots demonstrate a 5 display operation 2806. A user selection of a function 
graphical user interface that is icon based. However, other representation for the first device is received at input opera- 
forms of a user interface on screen 2112 are also suitable, tion 2808. 

such as a text-based user interface. Furthermore, many The device selections are redisplayed and the user selects 

variations on the graphical user interface shown are possible . a second device representation at input operation 2810. The 

FIG. 22 shows a screenshot 2200 that contains icons that 10 function representations of the second device are displayed 

form representations of the devices present within the envi- at display operation 2812. A user selection of a second 

ronment. As shown, six devices are present within the device selection for the second device is received at input 

environment and the screenshot 2200 includes a television operation 2814. The function representation selected for the 

representation 2202, a VCR representation 2204, a micro- first device is associated with the function representation for 

wave representation 2206, a stove/oven representation 2208, 15 the second device to create the interaction rule at rule 

a washer representation 2210, and a dryer representation operation 2816. 

2212. Also included in the screenshot 2200 are a rule button FIG. 24 shows a screenshot 2400 that is displayed after a 
2214, a first learn mode button 2216, and a second learn user selects the second learn mode button 2218. The screen- 
mode button 2218. s hot 2 400 includes a button 2402 that is a choice to learn a 

From screenshot 2200, the user may make a selection of 20 first portion of the interaction rule. Hie user presses the 

a device representation to leam information about the device button 2402 and then selects the first function on the first 

such as its current state. The logical operations of viewing device itself within the environment. In response to the first 

device information are discussed in FIG. 27. The selection device providing a message about its resulting state change, 

may also be used to send an instruction to the device to the selected function is displayed in field 2404. 

immediately bring about a state change as may be done with 25 The user then presses the button 2406 that is a choice to 

an ordinary remote control. The user may select the rule learn a second portion of the interaction rule. The user 

button 2214 to view interaction or translation rules already selects the second function on the second device itself, and 

stored and being executed for the environment. An example in response to the second device providing a message about 

of viewing existing rules is discussed in more detail with its state change, the selected function is displayed in field 

reference to FIG. 25. 2408. The interaction rule is created by associating the 

The user may also make a selection of the first leam mode function shown in the first display field 2404 with the 

button 2216 to program an interaction or translation rule by function shown in the second display field 2408. In the 

interacting with device representations and function repre- example shown, the rule that results is if the VCR is powered 

sentations for the device. The first leam mode is discussed 35 on (a first received state change message), then the TV tunes 

in more detail with reference to FIG. 23 and the logical to channel 3 (a second received state change message), 

operations of FIG. 28. Additionally, the user may make a The development of the rule may continue as well. The 

selection of the second learn mode button 2218 to program user may press the button 2410 that is a choice to learn a 

an interaction rule by interacting with the device itself. The third portion of the interaction rule. The user selects the third 

second learn mode is discussed in more detail with reference 4Q function on the third device itself, and in response to the 

to FIG. 24 and the logical operations of FIG. 26. third device providing a message about its state change, the 

FIG. 23 shows a screenshot 2300 after a user has selected selected function is displayed in filed 2412. In the example 

the TV representation 2202 from the initial screenshot 2200. shown, the rule that results is if the VCR is powered on, then 

The screenshot 2300 shows the device representation or icon the TV tunes to channel 3 and then the VCR begins to play. 

2302 and the associated function representations or icons for 45 Additionally, as discussed above in relation to advanced 

the functions of the TV present in the environment. The interaction rules of the aggregator, the user may specify via 

function representations include channel selection represen- buttons 2414, 2416 whether the execution of the multiple 

tation 2304, volume selection representation 2306, mute step interaction rule should be performed in parallel or serial 

representation 2308, power representation 2312, and signal fashion. If in parallel, then turning the VCR on causes 

input representation 2310. The user makes a selection of a 50 messages to be simultaneously instructing the TV to tune to 

particular function representation to be associated in an channel 3 and the VCR to begin playing simultaneously. If 

interaction rule and then selects another function represen- in series, then the TV is instructed to turn on prior to the 

tation of the TV or another device to complete the rule. VCR being instructed to play. 

As described above, the user may select a power on The logical operations of an example of the second learn 

representation for the VCR and then select the channel 55 mode are shown in FIG. 26. The logical operations begin at 

selection representation 2304 to indicate a channel 3 for the choice operation 2602 where the first choice is provided to 

TV. The interaction rule is created as a result so that the user for selection to initiate learning of the first portion 

whenever the VCR is powered on, the TV automatically of the interaction rule. The user selects the first choice, 

tunes to channel 3. The interaction rule may be programmed which is received at input operation 2604. The user then 

to include additional associations as well, such as setting the 60 selects the first function on the first device itself at input 

TV volume representation 2306 to a particular volume operation 2606. A state change message from the first device 

setting as well once the VCR is powered on. Likewise, rules is received at the device creating the rule at receive operation 

may be specified for a single device, such as for example 2608, and the message indicates the function the user 

specifying that when the TV is turned on, the volume of the selected. The function description is stored in memory. 

TV should automatically be set to a particular level. 65 The second choice is provided to the user for selection to 

The logical operations for the first learn mode are shown initiate learning of the second portion of the interaction rule 

in FIG. 28. The logical operations begin by the user interface at choice operation 2610. The user then selects the choice at 
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input operation 2612 to initiate learning of the second 
portion of the interaction rule. The user then selects the 
second function representation on the second device at input 
operation 2614. A state change message is received from the 
second device at the device creating the rule at receive 
operation 2616, and the message indicates the function the 
user selected. The function description is stored in memory. 
Once the two function descriptions are known by the device 
creating the rule, the first function description is associated 
with the second function description at rule operation 2618 
to create the reaction rule. 

FIG. 25 shows a screenshot 2500 that results from the user 
selecting the rule button 2214 and a selection of a device that 
is involved in the rule. For example, the user may select the 
TV representation 2202 to view an interaction rule involving 
the TV present in the device environment. A TV represen- 
tation 2502 is displayed and is connected to a function 
representation 2504 that indicates that the TV is being tuned 
to channel 3. A VCR representation 2506 is displayed and is 
connected to a function representation 2508 that indicates 
that the VCR is being powered on. 

A connector 2510 is shown connecting the VCR function 
representation 2508 to the TV function representation 2504. 
As shown, the connector 2510 is directional as an arrowhead 
points to the TV function representation 2504 to indicate that 
the TV function results from the VCR function. The corre- 
sponding interaction rule provides the association of VCR 
on to TV channel 3 only to automatically control the TV in 
response to the VCR but not the other way around. Thus, 
when the TV is tuned to channel 3 by the user, the VCR does 
not automatically turn on because the interaction rule is 
learned as a directional association. 

Other interaction rules may involve a connector that is not 
directional so that the association is absolute rather than 
directional. For example, it may be preferred that the VCR 
automatically turn on when the TV is tuned to channel 3 and 
that the TV automatically tune to channel 3 when the VCR 
is turned on. Such an interaction rule would be absolute 
rather than directional, and the connector 2510 would lack 
an arrowhead or alternatively have arrowheads pointing in 
both directions. One skilled in the art will recognize that 
other visual connectors besides arrowheads and lines are 
suitable as well. 

To view information about a specific device in the 
environment, the user may select the device representation 
from the screenshot 2200 of FIG. 22. A screenshot such as 
the screenshot 2300 of FIG. 23 will be displayed. Along side 
each function representation, the value for that function may 
be displayed to inform the user of the current state of the 
device. For example, a 3 may appear next to the channel 
representation 2304 while a checkmark appears next to the 
mute representation 2308 to indicate that the TV is currently 
tuned to channel 3 but is muted. 

The logical operations for obtaining information about a 
device through the device interface are shown in FIG. 27. 
The logical operations begin by displaying a choice of 
device representations at display operation 2702. A user 
selection of a device representation is received at input 
operation 2704 to select a first device. A message is then sent 
from the user interface, such as an aggregator, to the first 
device selected by the user at send operation 2706. The 
message includes a request for information, such as the 
current status, from the first device. 

The first device receives the request for information at 
receive operation 2708. The first device then directs a reply 
back to the user interface at send operation 2710. The reply 
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is a response to the request for information and includes the 
current status information of the first device. The user 
interface device receives the reply from the first device at- 
receive operation 2712, and then the user interface displays 

5 the current status information on the screen at display 
operation 2714. 

Various embodiments of devices and logical operations 
have been discussed above in relation to communications 
between devices, automatic and manual learning of interac- 

io tion rules, content control, aggregator functions, remote 
communications, and a user interface. Although these 
embodiments of devices and logical operations may be 
combined into a robust system of device interaction, it 
should be noted that various devices and logical operations 

15 described above may exist in conjunction with or indepen- 
dently of others. 

Although the present invention has been described in 
connection with various exemplary embodiments, those of 
ordinary skill in the art will understand that many modifi- 

20 cations can be made thereto within the scope of the claims 
that follow. Accordingly, it is not intended that the scope of 
the invention in any way be limited by the above description, 
but instead be determined entirely by reference to the claims 
that follow. 

25 What is claimed is: 

1. A method of establishing device interaction in an 
environment having an aggregator and a plurality of devices 
where each device performs one or more unique functions 
within the environment with the one or more functions of 

30 each device being associated with various states, the method 
comprising: 

transmitting a message to the aggregator from a first 
device; 

35 receiving the message at the aggregator; and 

referencing interaction rules at the aggregator in relation 
to the message from the first device to direct commu- 
nications from the aggregator to one or more devices of 
the plurality, wherein referencing interaction rules at 

40 the aggregator comprises detecting from the interaction 
rules whether to direct communications to multiple 
devices of the environment such that serial execution or 
parallel execution of state changes occur at the multiple 
devices. 

45 2. The method of claim 1, further comprising in response 
to referencing interaction rules at the aggregator, transmit- 
ting a message to multiple devices of the environment 
directing serial execution of state changes. 
3. An aggregator for establishing device interaction in an 
50 environment having a plurality of devices where each device 
performs one or more unique functions within the environ- 
ment with the one or more functions of each device being 
associated with various states, the aggregator comprising: 
at least one transmitter; 
55 at least one receiver; 

a memory that stores interaction rules for the plurality of 
devices; and 

a processor in communication with the transmitter, 
receiver, and memory, the processor being configured 

60 to receive a message from a first device through the at 
least one receiver, reference the interaction rules in 
relation to the message from the first device to direct 
communications through the at least one transmitter to 
one or more devices of the plurality, wherein referenc- 

65 ing interaction rules comprises detecting from the inter- 
action rules whether to direct communications to mul- 
tiple devices of the environment such that serial 
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execution or parallel execution of state changes occur 
at the multiple devices. 

4. The aggregator of claim 3, wherein the processor is 
further configured to transmit a message to multiple devices 
of the environment directing parallel execution of state 
changes in response to referencing interaction rules. 

5. An aggregator for establishing device interaction in an 
environment having a plurality of devices where each device 
performs one or more unique functions within the environ- 
ment with the one or more functions of each device being 
associated with various states, the aggregator comprising: 

a transmitter for each of at least two transports, such that 
the aggregator comprises a first transmitter for a first 
transport and a second transmitter for a second trans- 
port; 

a receiver for each of the at least two transports, such that 
the aggregator comprises a first receiver for the first 
transport and a second receiver for the second trans- 
port; 

a memory that stores interaction rules for the plurality of 
devices; and 

a processor in communication with the transmitters, 
receiver, and memory, the processor being configured 
to receive a message from a first device through the first 
receiver for the first transport, reference the interaction 
rules in relation to the message from the first device, 
translate the message from the first transport to the 
second transport, and send the translated message of 
the second transport through the second transmitter; 
and wherein the processor is further configured to 
receive a message through the second receiver, refer- 
ence the interaction rules, translate the message from 
the second transport to the first transport, and send the 
translated message through the first transmitter. 

6. A method of establishing device interaction in an 
environment having an aggregator and a plurality of devices 
where each device performs one or more unique functions 
within the environment with the one or more functions of 
each device being associated with various states, the method 
comprising: 

transmitting a message to the aggregator from a first 

device over a first transport; 
receiving the message at the aggregator; 



and the change of state from the message to find an 
associated change of state for a second device; and 
transmitting an instruction containing the associated 
change of state to the second device from the aggre- 
5 gator. 

8. A method of establishing device interaction in an 
environment having an aggregator and a plurality of devices 
where each device performs one or more unique functions 
within the environment with the one or more functions of 

10 each device being associated with various states, the method 
comprising: 

generating interaction rules at the aggregator, wherein 
generating the interaction rules comprises receiving at 
the aggregator a change of state message from a first 
device and a change of state message from a second 
device, and when the change of state messages from the 
first and second device are separated in time by less 
than a predefined amount, associating in memory at the 
aggregator the change of state of the first device with 
the change of state of the second device to form 
interaction rules; 
detecting a change of state at the first device; 
in response to detecting the change of state, transmitting 
a change of state message to the aggregator from the 
first device; 

receiving the change of state message at the aggregator; 
referencing interaction rules at the aggregator in relation 
to the change of state of the first device to determine a 
change of state instruction for the second device; and 
transmitting the change of state instruction to the second 
device from the aggregator. 

9. The method of claim 8, wherein generating the inter- 
action rules further comprises: 

receiving at the aggregator a change of state message 

from a third device; and 
when the change of state message from the third device is 
separated in time by less than the predefined amount 
from the message of the first or second device, asso- 
ciating in memory at the aggregator the change of state 
of the third device with the change of state of the first 
device and the change of state of the second device. 

10. An aggregator for establishing device interaction in an 
environment having a plurality of devices where each device 



15 



20 



25 



30 



35 



40 



referencing interaction rules at the aggregator in relation « performs one or more unique functions within the 



to the message from the first device to direct commu 
nications from the aggregator to one or more devices of 
the plurality; 

translating at the aggregator the message from the first 
transport to a second transport; and 

sending the message from the aggregator to one or more 
devices over the second transport. 

7. A method of establishing device interaction in an 
environment having an aggregator and a plurality of devices 
where each device performs one or more unique functions 
within the environment with the one or more functions of 
each device being associated with various states, the method 
comprising: 

transmitting a message to the aggregator from a first 
device indicating a change of state of the first device; 

receiving the message at the aggregator; 

referencing interaction rules at the aggregator in relation 
to the change of state message from the first device to 
direct communications from the aggregator to one or 65 
more devices of the plurality, wherein referencing 
interaction rules comprises looking-up the first device 
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environ- 
ment with the one or more functions of each device being 
associated with various states, the aggregator comprising: 
at least one transmitter; 
at least one receiver; 

a memory that stores interaction rules for the plurality of 
devices; and 

a processor in communication with the transmitter, 
receiver, and memory, the processor being configured 
to receive a message from a first device through the at 
least one receiver, reference the interaction rules in 
relation to the message from the first device to direct 
communications through the at least one transmitter to 
one or more devices of the plurality, and wherein the 
processor is further configured to receive through the at 
least one receiver change of state messages from the 
plurality of devices upon changes of state occurring at 
the plurality of devices to automatically create the 
interaction rules from associating the change of state 
messages. 

11. An aggregator for establishing device interaction in an 
environment having a plurality of devices where each device 



11/19/04, EAST Version: 2.0.1.4 



US 6,795,404 B2 



25 



26 



performs one or more unique functions within the environ- 
ment with the one or more functions of each device being 
associated with various states, and wherein one or more 
devices of the plurality maintain interaction rules, the aggre- 
gator comprising: 

at least one transmitter; 

at least one receiver; 

a memory that stores interaction rules for the plurality of 
devices; and 

a processor in communication with the transmitter, 
receiver, and memory, the processor being configured 
to send a message through the at least one transmitter 
to a device of the plurality that maintains interactions 
rules requesting the interaction rules of the device, 
receive the interaction rules sent by the device through 
the at least one receiver, and store the interaction rules 
in the memory, and wherein the processor is further 
configured to receive a message from a first device 
through the at least one receiver, reference the interac- 
tion rules stored in memory in relation to the message 
from the first device to direct communications through 
the at least one transmitter to one or more devices of the 
plurality. 

12. An aggregator for establishing device interaction in an 
environment having a plurality of devices where each device 
performs one or more unique functions within the environ- 
ment with the one or more functions of each device being 
associated with various states, the aggregator comprising: 

at least one transmitter; 
at least one receiver; 

a memory that stores interaction rules for the plurality of 
devices; and a processor in communication with the 
transmitter, receiver, and memory, the processor being 
configured to receive a message indicating a change of 
state from a first device through the at least one 
receiver, reference the interaction rules in relation to 
the message from the first device by looking-up the first 
device and the change of state from the message to find 
an associated change of state for a second device, and 
transmit an instruction containing the associated 
change of state to the second device through the at least 
one transmitter. 

13. An aggregator for establishing device interaction in an 
environment having a plurality of devices where each device 
performs one or more unique functions within the environ- 
ment with the one or more functions of each device being 
associated with various states, the aggregator comprising: 

at least one transmitter; 
at least one receiver; 
a memory that stores interaction rules for the plurality of 
devices; and 

a processor in communication with the transmitter, 
receiver, and memory, the processor being configured 
to receive through the at least one receiver a change of 55 
state message from the first device and a change of state 
message from a second device, and when the change of 
state messages from the first and second device are 
separated in time by less than a predefined amount, 
associate in the memory the change of state of the first 
device with the change of state of the second device to 
form interaction rules, and wherein the processor is 
further configured to receive a message from the first 
device through the at least one receiver, reference the 
interaction rules in relation to the message from the first 
device to direct communications through the at least 
one transmitter to the second device. 



10 



15 



20 



25 



30 



35 



40 



45 



50 



60 



65 



14. A method of establishing device interaction in an 
environment having an aggregator and a plurality of devices 
where each device performs one or more unique functions 
within the environment with the one or more functions of 
each device being associated with various states, the method 
comprising: 

transmitting a message to the aggregator from a first 
device; 

receiving the message at the aggregator; 

referencing interaction rules at the aggregator in relation 

to the message from the first device; 
in response to referencing the interaction rules, sending a 

message from the aggregator to a second device; 
in response to receiving the message from the aggregator, 

sending a message from the second device to the 

aggregator; 

receiving the message from the second device at the 

aggregator; and 
referencing the interaction rales ' at the aggregator in 

relation to the message from the second device to 

communicate with one or more devices of the plurality 

from the aggregator. 

15. The method of claim 14, wherein the message from 
the first device is a request of information from the second 
device and wherein the message from the second device is 
a reply with the information, and referencing the interaction 
rule after receiving the message from the second device 
causes the aggregator to transmit a message with the reply 
to the first device. 

16. A method of establishing device interaction in an 
environment having an aggregator and a plurality of devices 
where each device performs one or more unique functions 
within the environment with the one or more functions of 
each device being associated with various states, the method 
comprising: 

transmitting a message to the aggregator from a first 
device; 

receiving the message at the aggregator; 

referencing interaction rules at the aggregator in relation 
to the message from the first device to direct commu- 
nications from the aggregator to one or more devices of 
the plurality; 

receiving change of state messages from the plurality of 
devices upon changes of state occurring at the plurality 
of devices to automatically create the interaction rules 
from associating the change of state messages. 

17. A method of establishing device interaction in an 
environment having an aggregator and a plurality of devices 
where each device performs one or more unique functions 
within the environment with the one or more functions of 
each device being associated with various states, and 
wherein one or more devices of the plurality maintain 
interaction rules, the method comprising: 

transmitting a message to the aggregator from a first 
device; 

receiving the message at the aggregator; 

sending a message to a device of the plurality that 
maintains interactions rules, the message requesting the 
interaction rules of the device; 

receiving the message at the device that maintains inter- 
action rales; 

in response to receiving the message requesting the inter- 
action rules, transmitting the interaction rules from the 
device maintaining interaction rules to the aggregator; 
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receiving the interaction rules of the device at the aggre- 
gator; 5S 

storing the interaction rules in memory at the aggregator; 
and ^ 

referencing interaction rules at the aggregator in relation 
to the message from the first device to direct commu- 
nications from the aggregator to one or more devices of 
the plurality. 

18. A method of establishing device interaction in an 
environment having an aggregator and a plurality of devices 
where each device performs one or more unique functions 
within the environment with the one or more functions of 
each device being associated with various states, the method 
comprising: 

detecting a change of state at a first device; 
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in response to detecting the change of state, transmitting 
a change of state message to the aggregator from the 
first device; 

receiving the change of state message at the aggregator; 
referencing interaction rules at the aggregator in relation 

to the change of state of the first device to determine a 

change of state instruction for the first device and at 

least a second device of the plurality; 
transmitting the change of state instruction to the device 

from the aggregator; and 
in response to receiving the change of state instruction 

from the aggregator, sending a message from the device 

to the aggregator. 
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